OCT 3 1 2005 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

DECLARATION UNDER 37 C.F.R. $ 1.131 

1 . The following declaration is directed to establishing the date of the invention claimed in 
U.S. Patent Application No. 09/881 142, filed June 14, 2001. 

2. . To my knowledge, Paul M. Thomsen (the "named inventor") is the sole inventor of the 
claims in the above-referenced application. 

3. I am a partner at the law firm Knobbe Martens Olson and Bear. I am also a registered 
patent attorney and was responsible for preparing and filing the above-referenced patent 
application. 

4. From at least October 4, 2000 up until December 1, 2000, 1 actively worked in the normal 
course of my docket to finalize and file the above-referenced application. 

• During the period between October 4, 2000 and October 18, 2000 I conducted a 
patentability search for the invention. 

• On November 12, 2000, I received an Invention Disclosure Form providing a proposal 
of the invention and an amendment from Paul Thomsen. 

• On November 27, 2000, I provided a draft specification directed to the above case to 
Paul M. Thomsen, the inventor of the subject invention for review by him, and to Joseph D. 
Smallwood, Corporate Counsel for Hitachi Home Electronics. The November 27, 2000 
specification described the presently claimed invention. 

• On November 28, 2000, I conferred with Paul M. Thomsen regarding the November 
27, 2000 draft. 

• On December 1, 2000, I filed U.S. provisional application No. 60/250,835, entitled 
"System and Method for Selecting Text on Television Display." The '835 application is the 
priority application for the 6 142 application. 



Penalty of Perjury Statement 
I declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful, false statements and the like so made are 



punishable by fine or imprisonment, or both, under Section 10001 of Title 18 of the United 
States Code and that such willful, false statements may jeopardize the validity of the application 
or document or any patent resulting therefrom. 



Dated: October 27, 2005 By: 




Eric M. Nelson, Esq. 
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8 1 2005 &/ IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
f&B&V&s DECLARATION UNDER 37 C.F.R. $1.131 

1. The following declaration is directed to establishing the date of the invention claimed in 
U.S. Patent Application No. 09/881 142, filed June 14, 2001 . 

2. I, Paul M. Thomsen (the "named inventor") am the sole inventor of the claims in the 
above-referenced application. I was an employee of Hitachi Home Electronics while I invented 
the subject matter of the above-referenced application. 

3. I conceived of the invention as early as September 6, 2000, and 1 diligently worked in 
conjunction with in house and independent counsel up to file a provisional application on 
December 1 , 2000. 

4. From at least September 6, 2000 up until December 1, 2000, in this country T actively 
worked in the normal course of my workload to prepare and file the above-referenced 
application, 

• On or before September 6, 2000, 1 drew a sketch documenting aspects of the invention. 

• On or before September 6, 2000, I provided to Joseph D. Small wood (Corporate 
Counsel for Hitachi Home Electronics) an Invention Disclosure Form providing a proposal of the 
invention. 

• Prior to September 6, 2000, I provided to Joseph D. Smallwood an amendment To the 
proposal. See Attachment A, which is a copy of the proposal with dates and Hitachi and 3 rd 
party proprietary information redacted. 

• On November 12, 2000, I sent the proposal and the amendment to Eric Nelson, of 
Knobbc. Martens, Olson & Bear, LLP, Hitachi's outside patent counsel. 

• On November 27, 2000, I received a draft specification directed to the-above case from 
Eric Nelson, for review by me, and Joseph D. Smallwood, Corporate Counsel for Hitachi Home 
Electronics. The November 27, 2000 specification described the presently claimed invention. 

• On November 28, 2000, 1 conferred with Eric Nelson regarding, the November 27, 2000 

draft. 

• On December I, 2000, 1 filed provisional application 60/250,835, entitled "System and 
Method for Selecting Text on Television Display. 71 



Penalty of Perjury Statement 



I declare that all statements made herein of my own knowledge arc true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful, false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 10001 of Title 18 of the United 
States Code and that such willful, false statements may jeopardize the validity of the application 
or document or any patent resulting therefrom. 




Paul M. Thomsen 
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Joe Smallwood page 
K. Ohashi WITNESSED and UNDERSTOOD 



Thomsen 





FROM: 
DATE: 



SUBJECT: CAPTION SURFER PATENT PROPOSAL- 



f AMENDMENT 



Joe, 



we discussed it with HAL-Legal, David Cunningham. At your and his request 



Following yogr receipt of the original fflD9 patent P ro P osa ' for this concept, 

) ^nd^ ^mn 

attached is an amendment to that B B BR roposal with further, deeper 
details regarding the features and TundSons of Caption Surfer. I did not 
specifically copy David on this cover memo as I thought you would want to 
review first and then forward via y our offi ce to David's attention. Included as 
a separate pdf file is a copy of the JQproposal with two witnessed and 
understood signatures. 

In addition to the deeper detail discussion, I have also amended some of the itemized 
elements on our internal patent application form. Those itemized amendments 
immediately follow this cover page as page 2-7 + slides 8-22 in file CS_EXAM.ppt 
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Microsoft's TV PAK web access is a member only site requiring a login ID and 
password. When I emailed to Microsoft and asked how to become a member 
they responded with an email advising that since I'm with Hitachi ( based on my 
email address and information I supplied to them (work address, phone, etc.) I 
would be allowed access to their site. Microsoft's email included the ID and 
password so I was able to obtain the attached exhibit ( if s why I never asked you 
to approve an NDA with Microsoft for this subject! ). I have no idea how to 
handle such confidential documents in a patent disclosure application so I leave 
that to your and Davids expertise. 

Final point is a group of color Powerpoint slide printouts attached showing one 
example of how this concept would work from the customer's perspective. I also 
have the electronic Powerpoint presentation (including animations) and have 
supplied to you via email. 



AMENDMENT TO CAPTIj(jN^gER^| 
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4. Please explain background circumstances 



5. Please explain the objects, features. 



Original , proposal did not detail sufficiently the "object" ( purpose and 
operation ) of the invention. Further details are included as noted below. 

OBJECT: please see page 4 

TECHNICAL DESCRIPTION: please see pages 5-7 
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6. DESCRIPTION OF "BEST* MODE 



1? 



■PLEASE SEE SLIDES pgs 8 ~ 22 and ELECTRONIC EXAMPLE 
CS_.EXAM.ppt? 
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OBJECT of INVENTION: 

Purpose of invention is to allow TV viewer to locate information using keywords selected 
from U.S. closed caption data services commonly in use by today's television, movie 
studio ( VCR & DVD rentals ) or virtually any other data source supplying EIA-608 
compliant VBI data streams to a video display equipped with a closed caption decoder 
and this invention. 

The viewer can pick a word ( or words ) from the TV's on-screen closed caption display 
accompanying many television, VCR, DVD, satellite, cable or other program sources. 
The selection of the word(s) on the TV screen is accomplished using a "cursor" style or 
thumbstick or joystick type remote control. 

Once the word(s) has/have been selected, the invention then "finds" those words in the 
TV's closed caption display memory; converts them to a format suitable for input to any 
one of many Internet browsers ( or other data locating means ). The browser ( or other 
data locating means ) returns with data ( from the Internet or other data source ) that most 
closely match the keyword search terms "selected" by the customer from the closed 
caption on-screen display. 

The "information" ( as an easy to understand example ) could be from the Internet but 
this invention does not restrict itself to just the Internet The information source being 
accessed by this invention could also be local ( built-in to the TV ) or a corporate, or 
library-based source of information. 

The attached Powerpoint slides depict one example of this invention's purpose to locate 
information relevant to the video program being watched. These slides represent only 
one example and do not mean to limit the scope of the intended invention. 
Accompanying this paper description is a separate electronic file CS_EXAM.ppt of the 
Powerpoint slide presentation including animations showing this "typical" operation. 

Some other applications would include searching a "warehouse" of video or other 
material with closed caption data in order to find warehouse material matching the 
keyword or "search term(s)" submitted by the user. In this materially different 
application of the invention, the user would supply the search term or keyword(s) by 
keyboard input and then the invention would search each program's closed caption data 
for word(s) matching the user supplied keyword(s). * : 

The Powerpoint example shows an Internet browser ( Microsoft's IE 5 ) used as the 
"search engine" but that is only one of many ways to implement a search function of 
remote or local data sources or "warehouses". The selected word(s) from the closed 
caption data stream could be converted to support many different types of "search 
means" with appropriate interface. A further example of a DVD-ROM (or RAM ) 
"jukebox" conaining of several ( or a hundred ) DVD-ROM discs able to be brought "on- 
line" by mechanical or electronic means. If these DVDs contain closed caption data 
encoded in compliance with EIA-608, this invention can search and catalog them. 
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TECHNICAL DESCRIPTION OF INVENTION: 

This invention is built upon existing hardware and software common in today's television 
receivers called CLOSED CAPTION DECODING. There is a standard developed by the 
EIA and authorized by the FCC for this data service using line 21 of a television signal's 
vertical blanking interval (VBI). The applicable standard is EIA-608A ( currently being 
revised to B ). The applicable FCC authorization is provided in the Code of Federal 
Regulations (CFR 47.1 19) governing implementation of EIA-608 in U.S. TV receivers 
and U.S. program caption encoding by content providers. 

Anyone can tune to most any TV program today and encounter closed captions by 
enabling that feature on the TV ( the feature is mandated by the CFRs in U.S. TVs of 13" 
and larger since 1 993 ). It can be observed by watching such a TV and program, that . 
there already exists a means for encoding program captions. The standards ( EIA and 
CFRs ) noted above also provide for a minimum level of support with respect to character 
sets, colors and control codes to effect the on-screen location and appearance of the 
closed captions. The standards also stipulate the method of encoding these characters and 
control codes in a manner consistent from channel to channel ( among TV stations ) and 
among content providers. This consistency guarantees the uniform transmission and 
reception/decoding of captioned programs throughout the U.S. 

To support the standard, many semiconductor manufacturers have / are producing 
microcontrollers ( micons ) with the hardware and software to implement the decoding 
and display of closed captions in a TV receiver. These same micons also typically handle 
the infra-red (IR) commands issued by a viewer's hand held remote control. Many 
remote controls today use a "cursor" or joystick style of moving about on the TV's screen 
in order to allow the user (viewer) to turn on or off or otherwise adjust features built-in to 
the TV ( such as opening an on-screen menu to enable closed caption display ). 

This invention adds additional software to the TV's micon allowing it to provide the 
essential functions of the invention, namely: 

# enable the display of closed captions when instructed by the user 

# "freeze" the scrolling of on-screen captions when instructed by the user 

# present on-screen to the user a "cursor" or other means allowing the user to 
"point" to a word or words within the frozen closed caption display area on- 
screen 

# relate the on-screen selected word(s) to closed caption data stored in the micon's 
closed caption buffer memory 

9 identify from that relation the specific word(s) in the closed caption buffer 

# copy the selected buffer memory locations to another new buffer 

# strip away the control codes before/after the new buffer's contents 
0 output the new buffer's processed data to an appropriate interface 

# manipulate the interface's controls means to cause a search of that interface's 
connected data sources for content containing/matching the new buffer's content 

» display the result of the interface's search results in list or table form 
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TECHNICAL DESCRIPTION OF INVENTION: (con't) 

In block diagram forai below is a "typical" implementation today of the U.S. standard 
for closed caption decoding with a brief description next to each block. 

video slgnaJ cm-screen display 

containing closed— |p~ 



caption data applied 
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CCD BUFFER 



OSD OUTPUT 



^(OSD) graphics 
output to video 



to TVs on-board display 
microcontroller 

o VIDEO SIGNAL contains captions encoded per EIA-608 

o "SLICER" is micon hardware and software that locates line 21 in the video signal 

and removes or "slices" the data from that line. Sliced data is put in micon' s RAM 
o "PARSER" examines the data sliced from line 21 to determine if it is valid caption 

data ( transmitted on line 21 in packets called CC1 or CC2 or Tl or T2 ) 
o IF sliced data is valid caption or text data it is placed into micon RAM called the 

"CCD BUFFER" 

( there are OTHER types of data provided via line 21 such as "VCHIP" data which 
is NOT displayed on-screen to the viewer and NOT considered in this invention ) 
o "OSD OUTPUT" is hardware and software in the micon that is preprogrammed to 
understand the control codes embedded in the caption data stream and position 
the caption data accordingly on the TV's screen ( in a manner similar to the 
on-screen TV menu system ) 



• EIA-608 contains many control codes providing a rich set of options for 
presenting caption data on-screen such as: 

0 Positioning the black background caption "window" on-screen 
o POP ON or PAINT ON "window" styles 

o Number of rows to present and scroll attributes of the caption window 
0 Special characters to denote that music is being played as well as other 
special characters 

To implement this invention's essential function it needs to control the hardware and 
software mentioned above to: 

m cause the caption feature to be enabled ( if it is not already enabled ) 

• * "freeze" the contents of an on-screen caption window when the viewer sees a 
word they would like to select 

( this prevents subsequent caption data and control codes from overwriting or 
erasing the on-screen display as well as the contents of the CCD BUFFER ) 

• activate the TV's on-screen cursor "pointer" ( or generate one if not provided ) 

• move the on-screen pointer' in response to viewer's commands as indicated by 
keystrokes on the TV's remote control 

• provide a means ( a specific remote control key stroke ) to signal the invention 
that the viewer has positioned the pointer over a word of interest 

m review the CCD BUFFER'S contents and examine control codes and character 
data to determine based on screen pointer location which word in the buffer 
corresponds to the word "pointed to" by viewer 
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• copy that word to a new "working" buffer ( RAM ) in order to strip away any 
remaining control codes ( and in the event the viewer points to a special character 
such as the clef symbol for music ) advise the viewer accordingly via the TV's 
on-screen menu system that a search for xxxxxxx ( the selected word ) will be 
made ( or that a selection error has been made ) 

• output the working buffer's processed data to one of many "interfaces" to be 
coupled to an appropriate means for searching the connected database or 
"warehouse" ( Internet search or other source material ). 

( in this invention "interface" could mean to provide the selected word as data to a 
PC or other device via USB, IEEE1394, RS-232, 12C, wireless, cellular, EtherNet, 
HomePNA, etc., etc.) 

• display the interface's retailing result as a list or table for the user's review and 
possible further action 

( in the included Powerpoint example slides the "interface" is an Internet capable 
browser built-in to the TV but this is not the only "interface" possible in order to 
satisfy the intent of this invention; it is one of the simplest to understand- It is 
possible via additional software and or hardware to constuct a means of interface, 
result display and further action using the TV's own built-in menu system or 
another device's interface with appropriate application software and/or hardware 
to provide the result and further action functions ). 

From the above description it is obvious that this invention builds upon existing 
hardware and software within current television receivers today. By adding only 
software most TV's today would be able to provide this inventions functions except 
for the search interface and result interface functions ( since most TVs today do NOT 
have Internet browsers built-in ). 

While TV hardware and software is very arcane and specific to each TV 
manufacturer's design group, a fully functional implementation of this invention 
could be realized quickly on a PC-based platfonn by adding to ANY standard PC the 
following: 

, a TV tuner card which supports closed caption decoding 

9 application software which performs the functions of this invention by reading the 
TV tuner card's CCD BUFFER; allowing the customer to move the PC's mouse 
to select; performing the same processing of the CCD BUFFER'S data versus PC 
display mouse cursor location on-screen to identify the selected word; feeding 
that word ( after stripping away control codes, etc. ) to the PC's Internet browser. 

An additional feature of this invention is it's ability to "look" for URLs 
( http://www.xxx.yyy or www.xxx.yyy ) sent via the captioning service of line 21 . When 
found in the CCD buffer, these could be AUTOMATICALLY copied and stored as a list 
for the customer to review and use as search terms to his/her browser at a later time. 



